22102 - Connection Degraded

Alarm Group:

DIAM

Description:

Connection is only available for routing messages with a priority greater than or equal to the connection's congestion level. This alarm is generated when:

• Connection congestion when the DSR Tx sender buffer is at maximum capacity
• The connection's administrative state is enabled and the connection is in congestion. Requests and Answers continue to be received and processed from the peer over the connection, and attempts to send Answers to the peer still occur. The alarm is raised when one of the following occurs:
  • Connection's Operational Status transitions from available to degraded (connection has become congested or watchdog algorithm has failed)
  • Connection's Operational Status transitions from unavailable to degraded (connection has successfully completed the capabilities exchange and is performing connection proving)
• Connection egress message rate threshold has been crossed
• Diameter connection is in watchdog proving
• Diameter connection is in graceful disconnect
• Diameter peer signaled the remote is busy
• Diameter connection is in transport congestion

Severity:

Major

Instance:

<Connection Name>

HA Score:

Normal

Auto Clear Seconds:

0 (zero)

OID:

eagleXgDiameterFsmOpStateDegraded

Cause:

This alarm is raised when:

• Connection congestion when the DSR Tx sender buffer is at maximum capacity
• The connection's administrative state is enabled and the connection is in congestion. Requests and Answers will continue to be received and processed from the peer over the connection and attempts to send Answers to the peer will still occur. The alarm is raised when one of the following occurs:
  • Connection's Operational Status transitions from available to degraded (connection has become congested or watchdog algorithm has failed)
  • Connection's Operational Status transitions from unavailable to degraded (connection has successfully completed the capabilities exchange and is performing connection proving)
• Connection egress message rate threshold has been crossed
• Diameter connection is in watchdog proving
• Diameter connection is in graceful disconnect
• Diameter peer signaled that the remote is busy
• Diameter connection is in transport congestion

Diagnostic Information:

1. View the Connection Performance measurement report for the +/- 1 hour congestion event.
2. Examine the Log file by using these commands:
   • # date >> tcp_stat_<hostname>
   • # cat /proc/net/tcp >> tcp_stat_<hostname>
   • # sleep 1
   • # cat /proc/net/tcp >> tcp_stat_<hostname>
   • # sleep 1
   • # cat /proc/net/tcp >> tcp_stat_<hostname>
   • # sleep 1
   • # cat /proc/net/tcp >> tcp_stat_<hostname>
   • # date >> tcp_stat_<hostname>
3. Examine the output of the command, netstat -canp --tcp | grep <remote IP:Port for conn> for few minutes.
4. Examine the corresponding Rx buffer on the connection in question using this command: netstat -canp --tcp | grep <remote IP:Port for conn>. The RxBuffer value is configured using ConnectionCfget.
5. Examine the overall network statistics for other issues using the command, netstat -i.
6. Examine the overall network delay using the command ping.
7. View the software release information.

Recovery:

1. View the Connection Performance measurement report for the +/- 1 hour congestion event.
2. Examine the log file by using these commands:
   • # date >> tcp_stat_<hostname>
   • # cat /proc/net/tcp >> tcp_stat_<hostname>
   • # sleep 1
   • # cat /proc/net/tcp >> tcp_stat_<hostname>
   • # sleep 1
   • # cat /proc/net/tcp >> tcp_stat_<hostname>
   • # sleep 1
   • # cat /proc/net/tcp >> tcp_stat_<hostname>
   • # date >> tcp_stat_<hostname>
3. Examine the output of the command netstat -canp --tcp | grep <remote IP:Port for conn> for few minutes.
4. Examine the corresponding Rx buffer on the connection in question using this command:netstat -canp --tcp | grep <remote IP:Port for conn>. The RxBuffer value is configured using ConnectionCfget.
5. Examine the overall network statistics for other issues using the command netstat -i.
6. Examine the overall network delay using the command ping.
7. View the software release information.
8. Identify the most recent Connection Degraded event in the event log for the connection and use the Event's recovery steps to resolve the issue.
9. Have the peer vendor examined their receive buffer usage during the event; if it is 0, this means the received messages were processed quickly and messages were not often stored in the receive buffer. In this case, Egress Transport Congestion was due to the peer not processing the message quickly enough (verify by examining the peer's receive buffer), or there is some delay introduced in the network
10. If the problem persists, it is recommended to contact My Oracle Support.